format: text and Markdown files can be written in UTF-16 - #83
Merged
Conversation
TXT and MD gain two settings, shared in internal/format/textenc so the two
cannot describe the same thing differently: encoding (utf-8, utf-16le,
utf-16be) and bom. Both default to what these formats already wrote, and five
new golden values pin the encoded paths.
An odd number of bytes is refused in UTF-16 rather than rounded. A UTF-16 file
cut to an odd length is rejected by Python, by V8 and by .NET when each is
asked strictly, and repaired in silence by all three when it is not - so the
file this tool must never write is the one nobody could tell apart by looking.
Half of all sizes become unreachable and the refusal names the reachable size
either side of the one asked for.
The registry mechanism for that already existed: SmallestAccepted walks to a
fixed point and already handled PNG refusing a band above a size it accepts.
Single byte encodings are deliberately not offered. The filler vocabulary is
ASCII, so latin-1 output is byte for byte identical to UTF-8 - a setting that
would change nothing. They belong with content that is not English.
TXT and MD also gain the structural check they never had. The comment saying
they had nothing to be checked against stopped being true rather than being
wrong: a file claiming to be UTF-16LE is a claim somebody else's decoder can
settle. The checker is told the encoding rather than sniffing it, for the
reason the CSV dialect is told.
Two things this found that were not the job:
- TestEveryFormatSurvivesItsReferenceTool skipped the structural check for
any format with no reference tool, so the new checkers would have been
dead on arrival with the guard green and reporting them as covered. It was
invisible while the only formats without a reader were also the only ones
without a checker.
- a strict decode alone blesses a file with its byte order reversed, because
0x74 0x00 read the other way is a valid CJK character. What catches it is
asking what the characters are, and only TXT was asking.
Nine mutations, all caught. TestEveryTextFormatIsValidUTF8 keeps its name and
says out loud that its claim is the default path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Width had no caller outside textenc. A fresh export with nobody asking for it adds to a category this tree already counts, so it goes in as unexported and the mutation patterns naming it follow the rename. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
txtandmdgain two settings, shared ininternal/format/textenc:encoding(
utf-8,utf-16le,utf-16be) andbom. Both default to what these formatsalready wrote, so a recipe that says nothing gets the same bytes it got before -
pinned by five new golden values, with every one of the 57 existing hashes
untouched.
An odd size is refused, not rounded
Every character takes two bytes in UTF-16, so only an even number of bytes can be
a whole file. Measured on three readers, in three languages, on the same file cut
to an odd length:
TextDecoder(fatal:true)UnicodeEncoding(throwOnInvalidBytes)Get-ContentThe last row is the one that decided it: the cut file looks exactly like the
whole one. So half of all sizes become unreachable and the refusal names the
size either side of the one asked for. The registry mechanism for that already
existed -
SmallestAcceptedwalks to a fixed point and already handled PNGrefusing a band above a size it accepts.
Single byte encodings are deliberately absent
The filler vocabulary is ASCII, so a file written as latin-1 is byte for byte the
same file as UTF-8 - compared with
cmp. That would be a setting that changesnothing. They belong with content that is not English, which is a separate change
with its own golden bytes.
Two things this turned up that were not the job
TestEveryFormatSurvivesItsReferenceToolskipped the structural check for anyformat declaring no reference tool, so the new checkers would have been dead on
arrival - with the guard green and reporting both formats as covered. It was
invisible while the only formats without a reader were also the only ones
without a checker.
0x74 0x00read the other way is a valid CJK character. What catches it isasking what the characters are, and only
txtwas asking.txtandmdalso gain the structural check they never had. The comment sayingthey had nothing to check against stopped being true rather than being wrong, and
was rewritten with the reason rather than deleted. The checker is told the
encoding rather than sniffing it, for the reason the CSV dialect is told.
Verification
gofmt,vet,preflight --quickall 12 checks🤖 Generated with Claude Code